System and methods for controlling automatic scrolling of information on a display or screen

ABSTRACT

A system  10  for controlling the automatic scrolling of information on a computer display or screen  12  is disclosed. The system  10  generally includes a computer display or screen  12,  a computer system  14,  gimbaled sensor system  16  for following and tracking the position and movement of the user&#39;s head  18  and user&#39;s eye  20,  and a scroll activating interface algorithm using a neural network to find screen gaze coordinates implemented by the computer system  14  so that corresponding scrolling function is performed based upon the screen gaze coordinates of the user&#39;s eye  20  relative to a certain activation area(s) on the display or screen  12.  The gimbaled sensor system  16  contains a gimbaled platform  24  mounted at the top of the display or screen  12.  The gimbaled sensor system  16  tracks the user&#39;s  22  head  18  and eye  20,  allows the user to be free from any attachments while the gimbaled sensor system  16  is eye tracking, and still allows the user to freely move his or her head when the system  10  is in use. A method of controlling automatic scrolling of information on a display or screen  12  by a user  22  is also disclosed. The method generally includes the steps of finding a screen gaze coordinate  146  on the display or screen  12  of the user  22,  determining whether the screen gaze coordinate  146  is within at least one activated control region, and activating scrolling to provide a desired display of information when the gaze direction is within at least one activated control region. In one embodiment, the control regions are defined as upper control region  208,  lower region  210,  right region  212,  and left region  214  for controlling the scrolling respectively in the downward, upward, leftward, and rightward directions. In another embodiment, the control regions are defined by concentric rings  306, 308,  and  310  for maintaining the stationary position of the information or controlling the scrolling of the information towards the center of the display or screen  12.

FIELD OF THE INVENTION

[0001] This invention relates to a computer interface system and methodsand their method of use, and, more particularly, to a computer interfacesystem and methods for controlling automatic scrolling of information ona display or screen.

BACKGROUND OF THE INVENTION

[0002] Prior Conventional Computer Mouse Devices Computer interfacessuch as mouses, track balls, light pens, etc. are devices that provideusers with a way of controlling and manipulating the display ofinformation, data, text, and images on a computer screen or monitor.Computer mouses have become a common and widely used device of presentcomputer systems. It is well known that a mouse typically has a rollerball placed inside a hand-held housing wherein the ball rolls along asurface and the ball also rolls against directional control contacts toallow a user to move a cursor within a screen or monitor. The housingalso usually has buttons on the top side for the user to activate userselection or to manipulate the display of information, data, text,images, etc. However, mouses require the use of hand or manualmanipulation by the user and have the disadvantages in that they requirethe use of a hand in order to be controlled. The disadvantages becomeevident when the user is in an environment which requires thesimultaneous use of hands for other purposes (i.e typing on a keyboard)or requires a hands-free environment or the user is physicallychallenged or handicapped as to the use of his or her hands.

[0003] Prior Eye Mouse Devices With Eye Cursor Control and Manual orFoot Activation or Selection

[0004] Eye controlled devices have been developed to control themovement of a cursor on a computer screen or display. These devicesallow the user to move the cursor by moving his or her eyes, and thesedevices accomplish the movement of the cursor by tracking the movementof the eyes of the user. However, some of these devices still requiremanual or foot activation or selection to control the display ofinformation, text, images, data, etc. An example of an eye controlleddevice with manual or foot activation or selection is seen in U.S. Pat.No. 5,367,315. These type of devices still have the main disadvantage inthat they require the use of a user's hand in order to be controlled.Therefore, these devices are still not well suited for users thatrequire a hands-free environment nor are they well suited to those userswho are physically challenged.

[0005] Prior Eye Mouse Devices With Eye Cursor Control and Dwell or GazeTime Activation or Selection

[0006] Eye mouse devices have been further developed so that themovement of a cursor on a computer screen or display is controlled bythe movement of the user's eyes and, furthermore, activation orselection is accomplished by the user dwelling, gazing, or staring at adesired activation region for a pre-determined amount of time. Forexample, this type of device is controlled by the user moving his or hereyes to a desired area on the computer screen or display and the userstares, dwells, or gazes at an activation region for a pre-determinedamount of time to activate menus or make selections or open windows,etc. Examples of eye mouse devices that have the features of an eyecursor control and dwell or gaze time activation or selection are seenin U.S. Pat. Nos. 4,836,670; 4,950,069; 4,973,149 and 5,345,281. Thistype of device allows a user to control both the movement of the cursorand the activation or selection of menus, windows, text, data,information, or images. However, in order to control the scrolling ofinformation, data, text, or images on the display or screen with thistype of device, the user will need to dwell, stare, or gaze at a scrollbar, page up or pace down activation region, etc. The problem with thistype of control of scrolling is that it is slow and requires the user towait at least the pre-determined amount of time before scrolling isactivated and implemented. Therefore, there is a need for and it is anobject of the present invention to provide a device for controlling viathe user's eye(s) the automatic scrolling of information, data, images,text, etc. on a computer screen or display.

[0007] Prior Eye Tracking Devices (Video or Image Processing)

[0008] It is well known that video or image processing has been used totrack the movement of the user's eye(s) for the purposes of controllingthe cursor on a computer display or screen. Examples of this type ofvideo or image processing for eye tracking purposes are demonstrated inU.S. Pat. Nos. 4,648,052; 5,331,149; 5,471,542; 4,513,317 and 5,481,622.One of the problems with the eye tracking devices for eye controlledcomputer devices is that normally a mechanism is required to be attachedto the user in order to track the movement of the user's eye(s). Users,however, typically prefer not to have attachments on them in order tooperate a computer peripheral since these attachments are usuallyuncomfortable to wear or they are not aesthetically pleasing,fashionable or ergonomic.

[0009] Non-attached eye tracking devices for controlling a cursor on adisplay or screen have been developed. An example of such a non-attacheddevice is seen in the prior art systems developed by LC Technologies atthe World Wide Web site http://www.lctinc.com/doc/ecs.htm. However, aproblem with these non-attached eye tracking devices is that they onlyallow or restrict the movement of the user's head within a limited rangesince the hardware has to be directed or aimed within the range orvicinity of the user's eye.

[0010] Therefore, there is a need for a device that tracks the eye ofthe user for allowing the control of a cursor on a display or screen andthat does not restrict or limit the movement of the user's head, whenthe device is not in anyway attached to the user.

[0011] Manipulation of Three Dimensional Images on a Screen or Display

[0012] It is also known that devices which track the position of the eyeof a user and that use the corresponding positional signals to controlthree dimensional images on a computer, video game, or other apparatusexist in the prior art. These devices use eye tracking to control thethree dimensional imagery and allows the user via eye movement and/ordwell time to jump levels, move to another display via selection menusand/or buttons on a display screen, move virtual objects forwards andbackwards, etc. Examples of these types of devices are seen in U.S. Pat.Nos. 5,293,187; 5,422,689 and 5,491,492. However, these devices also donot in any way teach or suggest the control of automatic scrolling of acomputer display or screen.

[0013] The Need for Automatic Scrolling Devices

[0014] Many applications and uses exist that require a user to be in ahands-free environment or in which the user does not have the benefit ofthe use of his or her hands (i.e. physically challenged, handicapped,etc.). Furthermore, scrolling of text, data, images, information, etc.on a display or screen is inherently needed for word processing, readinginformation (i.e. CD-ROM books), performing textual searches, viewingimages, observing real time data (i.e. air traffic control, satelliteweather pictures, etc.), etc. For example, users of word processingsoftware for a computer system need to scroll the text line by line orpage by page in order to read, scan, manipulate, or edit the text, data,information, etc. on the display or screen. Another example is a medicaldoctor who is performing medical procedure(s) or surgery and who wouldbenefit from having a device that allows him or her to continue themedical or surgical procedure (i.e. keep his or her hands on theprocedure) and can manipulate and scroll images or text (i.e. view CATScan, X-Ray Images, Patient Charts or Files; reference Medical TextBooks; etc.) or control the field of view and magnification of animaging device on a heads-up display or screen with the use of thedoctor's eyes. (See Lemelson and Hiett application entitled “ASELECTIVELY CONTROLLABLE HEADS-UP DISPLAY SYSTEM AND METHOD FORPRESENTING DESIRED MEDICAL INFORMATION AND/OR IMAGE(S)”, Ser. No.08/720,662; Filing Date Oct. 2, 1996). Further examples are automobilemechanics that are performing work on an automobile or an electricaltechnician or assembly-line worker that is working on a circuit board orother product or apparatus who may need to simultaneously or brieflyrefer to a manual, special instructions, or other pertinent informationduring the performance of their work wherein this information or datacan be accessed on a computer display or screen. Therefore, the need fora device for controlling the automatic scrolling of information, data,images, text, etc. on computer display or screen that allows the user tohave his or her hands free to do other desired tasks, and it is anobject of the present invention to provide such a device.

[0015] Notwithstanding the large number of articles and patents issuedin the area of eye mouses or eye controlled interfaces for computersystems, there has been no such device that is not highly restrained, ifunattached by the position of the user's head for providing automaticcontrol of scrolling of the information, data, or display, especiallythe automatic eye control of scrolling of information, data, or display.The device can be designed for the special needs of individuals thatrequire a hands-free environment or who are physically challenged orhandicapped. Such a device would be extremely useful to personnelworking in the fields of medicine, assembly lines, automobile servicestations, electronics assembly, or any other environments that requirethe performance of manual procedures and also have to simultaneouslyreference information data, information, images, text, etc.

[0016] Presently, there is a need for a hands free eye controlledscrolling device for computer systems. There is a further need toprovide an automatic scroll control device for automatically scrollingthe display of information, text, data, images, etc. on a computerdisplay or screen to provide a hands-free environment resulting inconvenience and efficient access of related information to the user.

OBJECTS OF THE INVENTION

[0017] One object of this invention is a system for controllingautomatic scrolling of information on a display or a screen thatincludes a computer system coupled to the display or the screen, an eye,head tracking, and/or speech scroll control sensor system coupled to thecomputer system for tracking and determining a gaze direction of the eyeof a user relative to the display or the screen, and a gimbaled sensorsystem coupled to an interface card attached to a computer system forimplementing automatic scrolling based upon the gaze direction of theeye of the user relative to an activation area on the display or thescreen.

[0018] Another object of this invention is the gimbaled sensor systemhaving an eye tracking system for tracking the eye of the user and aneye gaze direction determining system for determining the gaze directionof the eye of the user relative to the display or the screen.

[0019] Another object of this invention is that the gimbaled sensorsystem is coupled to a computer system for tracking and determining theposition of the eye of a user and a position of the head of the userrelative to the display or the screen.

[0020] A further object of this invention is the gimbaled sensor systemhaving a camera or photo sensor for providing images of the head and theeye of the user, a zoom lens coupled to the camera or photo sensor forfocusing the camera or photo sensor at the user, and optics coupled tothe camera or photo sensor for aiding the camera or photo sensor indetecting or providing images of the head and the eye of the user sothat the images can be processed by the computer system.

[0021] Another object of the invention is to allow the control ofinformation on a display or a screen by a user that includes the stepsof finding a gaze direction on the display or the screen of the user,determining the screen or display coordinates of which the user isfocused on and whether the coordinates are within at least one controlregion, and thereby activating scrolling to provide a desired display ofinformation when the screen gaze direction is within the at least oneactivated control region.

[0022] Another object of this invention is that the step of finding agaze screen coordinate on the display or the screen surface furtherincludes the steps of detecting a user that is closest to the display orthe screen, focusing and magnifying a field of view of a camera on aneye of the user to provide a magnified image of the eye, and reading themagnified image into a computer system, determining physical coordinatesof a center of a cornea of the eye and a glint (projected by a lightsource from gimbaled sensor system) center of the eye, determining avector between the center of the cornea and a glint center on the sensorimage, calculating screen gaze coordinates of the user, and sending thescreen gaze coordinates to the computer system for processing by anapplication program for controlling the scrolling or selection ofinformation on the display or the screen.

[0023] Another object of this invention is that the step of focusing andmagnifying a field of view of a camera on an eye of the user furtherincludes the steps of determining a correction vector for re-centeringthe camera to adjust the field of view and zooming the field of view ofthe camera onto the center of an eye of the user.

[0024] A still further object of this invention is that the step ofdetermining whether the screen gaze coordinates is within at least oneactivated control region further includes the steps of determiningwhether the screen gaze coordinates is within an upper horizontalrectangular region, a lower horizontal rectangular region, a rightvertical rectangular region, a left vertical rectangular region, orwithin an overlapping area of two regions, and scrolling the informationrespectively downwards, upwards, leftwards, rightwards, or in thecorresponding two directions, depending on the screen gaze coordinatesand scrolling the information at a rate defined by the screen gazecoordinates.

[0025] Another object of this invention is that the step of determiningwhether the screen gaze coordinates is within at least one activatedcontrol region further includes the steps of determining whether thescreen gaze coordinates is within a static region defined by at leastone of a number of concentric circles, determining whether the gazedirection is within an activated control region defined by another ofthe number of concentric circles, and activating scrolling to provide adesired display of information so that the region at which the screengaze coordinates of the user is directed is moved to a center of thedisplay or the screen at a rate that is defined for the concentriccircle at which the screen gaze coordinates of the user is directed.

[0026] A still further object of this invention is that the step ofdetermining whether the screen gaze coordinates is within an activatedcontrol region defined by another of the number of concentric circlesfurther includes the steps of calculating a radius and an angle withrespect to a center of the number of concentric circles to define ascreen gaze vector, calculating horizontal and vertical scroll ratesbased on the gaze vector, and scrolling the information on the displayor the screen in the horizontal and the vertical directions based on thecalculated horizontal and vertical scroll rates.

[0027] A further object of this invention is that the step ofdetermining whether the screen gaze coordinate is within an activatedcontrol region defined by another of the number of concentric circlesfurther includes the steps of determining whether the gaze direction iswithin an activated control quadrant wherein the information on thedisplay or the screen is moved downward and leftward; or upward andleftward; or upward and rightward; or downward and rightward.

[0028] A further object of this invention is a method of making a systemfor controlling automatic scrolling of information on a display or ascreen that includes the steps of providing a computer system coupled tothe display or the screen, coupling a gimbaled sensor system to acomputer system for tracking and determining the eye gaze direction of auser relative to the display or the screen, and coupling a scrollactivating interface system to the computer system and interfacing thescroll activating interface system with the gimbaled sensor system forimplementing automatic scrolling based upon the calculated screen gazecoordinates of the eye of the user relative to an activation area on thedisplay or the screen.

[0029] A further object of this invention is the step of coupling an eyescroll control sensor further includes the step of coupling an eye andhead scroll control sensor to the computer system for tracking anddetermining the position of the head of the user relative to the displayor the screen.

[0030] A further object of this invention is that the step of coupling agimbaled sensor system further includes the step of coupling a headsensor to the computer system for tracking the position of the head ofthe user relative to the display or the screen.

[0031] A further of this invention is to provide a system forcontrolling the automatic scrolling of information on a display or ascreen that allows the user to be able to freely use his or her hands toperform other tasks, procedures, work, etc.

[0032] Another object of this invention is to provide a system forcontrolling the automatic scrolling of information on a display or ascreen that tracks the user's eye, allows the user to be free from anyattachments while the system is eye tracking, and that still allows theuser to freely move his or her head when the system is in use. A furtherobject of this invention is to provide a system for controlling theautomatic scrolling of information on a display or a screen that can beused in various and wide applications such as for medical personnel,technicians, assembly line workers, weather persons, air trafficcontrollers, etc.

[0033] Further objects, features, and advantages of the invention willbecome evident in light of the following detailed description consideredin conjunction with the referenced drawings of a preferred exemplaryembodiment according to the present invention.

BRIEF DESCRIPTION OF FIGURES

[0034]FIG. 1A depicts a side view of an example system for controllingthe automatic scrolling of information on a computer display or screen.

[0035]FIG. 1B depicts a side view of an example system that is used witha head mounted (attached) device or glasses worn by the user wherein theuser is able to perform tasks that require the use of hands (i.e. atechnician working on a circuit board).

[0036]FIG. 1C depicts an automatic scrolling control system that is usedwith a non-attached screen or display of which may be transparent andwherein the screen or display provides hands-free, heads-up medicalinformation to the user (i.e. the control of scrolling of medicalheads-up display or screen information).

[0037]FIG. 2 is a general flow chart of the algorithm for screen gazecontrol that is implemented by the hardware (see FIGS. 3A, 3B, 3C).

[0038]FIG. 3A is a general block diagram of the eye control computerhardware.

[0039]FIG. 3B is a block diagram of a specific embodiment hardwaresystem for the screen gaze control system where the sensor is gimbaledto follow users head.

[0040]FIG. 3C is a block diagram of a more specific eye control systemhardware for tracking the user's head and eye wherein the system uses ahead/eye gimbaled sensor system which include pan/tilt servos foraccomplishing the head tracking.

[0041]FIG. 4A depicts a perspective view of the vectors and 3-D geometryin Euclidian coordinates that are associated with head/eye tracking.

[0042]FIG. 4B depicts a front field of view from a camera or photosensor used with the eye control system wherein images of three usersare detected or sensed by the camera or photo sensor and the images areused to determine the closest user to the display or screen.

[0043]FIG. 4C depicts a front magnified field of view from a camera orimage sensor used with the eye control system wherein images of the eyeof the user is detected or sensed by the camera or image sensor and theimages are used to determine the screen gaze coordinates of the user onthe display.

[0044]FIG. 5A is a flow chart of an image processing algorithm fortracking the user's head and eye.

[0045]FIG. 5B is a flow chart of a specific algorithm for screen gazecontrol that is implemented by the hardware (see FIGS. 3A, 3B, 3C).

[0046]FIG. 5C is an example of a neural network used for calculating thescreen gaze coordinates.

[0047]FIG. 6A depicts a front view of a display or screen showing apreferred embodiment for controlling automatic scrolling of information,images, or text wherein the preferred embodiment uses rectangularautomatic scroll control regions and scroll rate control regions forautomatically scrolling the display or screen.

[0048]FIG. 6B depicts a front view of a display or screen that is usedfor scrolling information, images, or text at which the user is gazingat the bottom right rectangular control region of the display or screen.

[0049]FIG. 6C depicts a front view of a display or screen that has theinformation, images, or text in the display or screen of FIG. 6Bscrolled upwards and to the left.

[0050]FIG. 6D is a block diagram of a specific algorithm for automaticscrolling of information, images, or text on a display or screen usingthe preferred embodiment of rectangular automatic scroll controlregions.

[0051]FIG. 7A depicts a front view of a display or screen showinganother preferred embodiment for controlling automatic scrolling ofinformation, images, or text wherein the preferred embodiment usesconcentric rings regions or a screen gaze position vector with respectto screen center for scroll rate control regions to automatically scrollthe display.

[0052]FIG. 7B depicts a front view of a display that is used forscrolling information, images, or text at which the user is gazing atthe bottom right quadrant control region of the display (i.e. usergazing at Florida at the bottom right corner of the United States map).

[0053]FIG. 7C depicts a front view of a display that has theinformation, images, or text in the display of FIG. 7B scrolling towardsthe center of the screen or display (i.e. image of Florida is moving tothe center of the display).

[0054]FIG. 7D is a block diagram of a specific algorithm for automaticscrolling or deleting of information, images, or text on a display orscreen using the other preferred embodiment screen gaze position vectorfor automatic scroll control.

DETAILED DESCRIPTION OF THE INVENTION

[0055] The present invention of this application is better understood inconjunction with the following detailed description of the Figures andthe preferred embodiment. The various hardware and software elementsused to carry out the invention are illustrated in the attached drawingsin the form of flow charts and block diagrams. For simplicity andbrevity, the Figures, and Specification do not address in detailfeatures that are well known in the prior art, such as the literaturelisted in the Background of the Invention above and certain additionalprior art which is discussed in the Detailed Description that follows.However, to assure an adequate disclosure, the specification herebyincorporates by reference each and every patent and other publicationreferenced above in the Background of the Invention or mentioned in theDetailed Description below.

[0056] I. Overall System and Applications of System

[0057] A. Automatic Scrolling for a Computer Display or Screen

[0058] In a preferred form of the invention as seen in FIG. 1A, a humanmachine interface system 10 for controlling the automatic scrolling ofinformation on a computer display or screen 12 is shown. The humanmachine interface system 10 generally includes a computer display orscreen 12, a computer system 14, gimbaled sensor system 16 for followingand tracking the position and movement of the user's head 18 and theuser's eye 20, and a scroll activating interface algorithm implementedby the computer system 14 so that corresponding scrolling function isperformed based upon the gaze direction of the user's eye 20 used tocalculate screen gaze coordinates relative to a certain activationarea(s) on the display or screen 12. In FIG. 1A, the user 22 sits infront of the computer display or screen 12, and the user 22 stares atcertain area(s) on the display or screen 12.

[0059] B. Applications for Automatic Scrolling

[0060]FIG. 1A shows that the automatic scrolling control system 10 isused for controlling information on a computer display or screen 12.FIG. 1B shows that the automatic scrolling control system 10 is usedwith a head mounted display or pair of display glasses 32. The headmounted device or glasses 32 is worn by the user 22, and the user 22 isable to perform tasks that require his/her hands 44 (i.e. FIG. 1Bspecifically shows user 22 working on a circuit board 48 while stillable to control the display of information with their eye 20), The headmounted display 32 can also be a medical heads up display as shown inFIG. 1C wherein a doctor, shown as the user 22, performs a medical orsurgical procedure on a patient 46 while still able to control thedisplay of information on the display or screen 42 with the user's eye20 (i.e. not shown, but see Lemelson and Hiett application entitled “ASELECTIVELY CONTROLLABLE HEADS-UP DISPLAY SYSTEM AND METHOD FORPRESENTING DESIRED MEDICAL INFORMATION AND/OR IMAGE(S)”, Ser. No.08/720,662; Filing Date Oct. 2, 1996). FIG. 1C also shows the automaticscrolling control system 10 used with a non-attached screen or display42 wherein the screen or display 42 provides heads-up display of medicalinformation while the user 22 (i.e. medical doctor) performs a medicalor surgical procedure on a patient 46. However, there are many otherapplications that exist which the present application may be utilized,and the present invention is not limited to the applications, ways, ormethods of implementation that the present invention is used as shown inFIGS. 1A, 1B, and 1C.

[0061]FIG. 1A shows that the head and eye tracking gimbaled sensorsystem 16 is not in any way attached to the user 22. The head and eyetracking system 10 includes a gimbaled sensor system 16 which includes apositionally-controlled, gimbaled platform 24 that is shown mounted atthe top of the display or screen 12. The gimbaled sensor system 16senses the position of the user's head 18 via a sensor view path 28 asshown in FIG. 1A, and the gimbaled platform 24 moves the gimbaled sensorsystem 16 with camera(s) 26 according to the position of the user's head18 and centers the focus of the gimbaled sensor system 16 with camera 26generally onto the user's eye 20 via camera line of sight 30. Thecentering of the focus of the camera(s) 26 can be achieved by imageprocessing, facial recognition, or any other suitable method. One suchmethod of head tracking is disclosed in Rekimoto, “A Vision-Based HeadTracker For Fishtank Virtual Reality: VR Without Head Gear”, IEEEVirtual Reality Annual international Symposium '95 (VRAS '95), 1995. Aprocess such as image processing, pupil recognition, or any othersuitable method is further used to determine the position at which theuser's eye 20 is focused onto the display or screen 12. However, anysuitable device that can track the movements of the head and eye of theuser may be utilized for carrying out the purposes of the presentinvention which is not limited to the ways shown in FIG. 1A.

[0062] B. Automatic Scrolling for a Head-Mounted Device

[0063]FIG. 1B shows another embodiment of the present human machineinterface system 10 in the form of a transparent screen or display 32that is mounted to the head 18 of the user 22 shown as an electronictechnician. The head-mounted screen or display 32 is in the shape ofglasses that is mounted over the user's eyes 20, and it has eye trackingsensors 34 and a microphone 36 mounted on one side of the glasses and aradio antennae 38 mounted on the opposite side of glasses. Thehead-mounted display or screen 32 provides a hands-free environment forthe user 22 to perform other tasks with hands 44 free while allowing theuser 22 to control the scrolling of information on the screen or display32 with the use of the user's eye(s) 20 and/or in combination with voicecommand. The head-mounted screen or display 32 is coupled via wirelesscommunication with computer system 14 by head mounted antenna 38 andcomputer mounted antennae 40.

[0064] However, any suitable devices, components, or apparatuses thatmount to the user's head 18 may be utilized for carrying out thepurposes of the present invention which is not limited to the ways shownin FIG. 1B.

[0065] One method of providing the eye tracking function for thehead-mounted display or screen 32 is by the use of a low power laserthat generates an infrared eye-tracking laser beam. The laser beam isprojected through a lens and reflected by a mirror onto the user's eye20. The user's eye 20 include a sclera, cornea, and pupil. When theuser's eye 20 move, the eye components cause distortions in the infraredlaser beam, which are reflected back onto mirror, and then through alens into an infrared photo detector, infrared camera, or other type ofphoto detector. This distortion of the laser beam corresponds to the eyedirection vector which can be measured accurately by eye-trackingelectronics. Data defining the eye direction vector is subsequentlytransmitted from the eye-tracking electronics to the command computer 14through wireless communication (i.e. computer transceiver antenna 40 and38 head mounted antenna). However, any suitable devices, components, orapparatuses for tracking the movements of the user's eyes 20 may beutilized for carrying out the purposes of the present invention which isnot limited to the ways shown in FIG. 1B.

[0066] C. Automatic Scrolling for Transparent, Non-Attached Screen

[0067]FIG. 1C shows a further embodiment of the present human machineinterface system 10 in the form of an optionally transparent,non-attached display or screen 42. The non-attached display or screen 42is shown to be a flat transparent screen that is in front of the user22. The display or screen 42 is attached to a computer 14 through cable13, and head and eye tracking system 16 with camera 26 interface betweenthe computer 14 and the user 22 to control the scrolling of theinformation on the display or screen 42. The head and eye trackingsystem 16 can be the same system as described for the embodiment of thepresent invention in FIG. 1A. However, any suitable display or screenfor displaying and controlling the display of information may beutilized for carrying out the purposes of the present invention which isnot limited to the ways shown in FIG. 1C. The display or screenmentioned above as a head mounted display screen 32 of FIG. 1B andnon-attached display screen 42 of FIG. 1C are considered the same outputdisplay device 12 throughout the rest of the detailed description ofthis invention. The display device 12 may also be described as a screen12 or display screen 12 which is used interchangeably throughout thisdocument and is intended to have the same meaning. The display 12 may bea part of a multimedia system which is capable of producing video,making sounds, or other inputs to any of the senses of the user 22.

[0068] II. The General Automatic Scrolling System

[0069] A. The General Software Algorithm for the Automatic ScrollingSystem

[0070]FIG. 2 shows the block diagram 56 of the general algorithm (i.e.software algorithm) to control the automatic scrolling of information ordisplaying on a display 12. The algorithm includes the step at block 58for obtaining sensor gaze coordinates from the user 22 but may also beused to find a gaze direction outside of the screen display but in thesame plane as the display surface. At decision block 60, thedetermination is made as to whether screen gaze coordinates are within acontrol region (i.e. on the border or peripheral region or on a menuswitch control). The control region may provide the user with thefurther feature of a smooth scrolling function (i.e. speed of scrollingdepends on position at which the gaze of the user is on the activationregion) or other type of menu selection function or feature. If the gazemeasurements are not within a control region, then further gazemeasurements are obtained from the user 22. If gaze measurements arewithin a control region, then the human-machine interface system 10 isactivated to scroll the information to the corresponding location on thedisplay 12. However, any suitable algorithm for controlling theautomatic scrolling of information on a display or screen may beutilized for carrying out the purposes of the present invention which isnot limited to the ways shown in FIG. 2.

[0071] B. The General Hardware for the Automatic Scrolling System

[0072]FIG. 3A shows a block diagram of the general hardware for theautomatic scrolling control system 10. The block diagram includes acomputer system 14, a display system represented at block 52 (i.e.display systems 12, 32, or 42) coupled to the computer system 14, and aneye/speech scroll control sensor at block 15. The user(s) 22 interfaceswith the display system 52 wherein the user 22 is in front of thedisplay system 52 and the user 22 is glancing at the display system 52.The sensor system 15 which may be free to move on a gimbaled platform 24detects the position of the user's head 18 and eye 20, and the gimbaledsensor system 54 interfaces with the computer system 14 to determine theposition on the display system 52 at which the user 22 is gazing. Thecomputer system 14 determines whether the user 22 is gazing at a regionfor activating automatic scrolling of information on the display system52 (this gaze region may be on the displays 12, 32, or 42 of FIGS. 1A,1B, and 1C). Also shown in FIG. 3A are other input devices 53 (i.e. keyboard, light pen, hand mouse, buttons, joysticks, etc.) for the user 22to interface with the computer system 14. A LAN or WAN 51 allows thecomputer system 14 to access more information outside of the system fordisplaying on the display 12.

[0073] However, any suitable hardware and/or software components may beutilized for carrying out the purposes of the present invention which isnot limited to the ways shown in FIG. 3A.

[0074] III. Detailed Disclosure of a Specific Automatic Scrolling System

[0075] A. General Hardware for a Gimbaled Sensor System Screen GazeCoordination for Automatic Scrolling System

[0076]FIG. 3B shows a specific automatic scrolling system hardware 10used for eye and head tracking that includes the use of a gimbaledsensor system 16 and pan/tilt servos represented at block 171 (i.e. panand tilt servos 94 and 98 shown in FIG. 3C) that move the sensor(s) 15that are components for the gimbaled sensor system 16. The computersystem 14 is shown interfaced with the display system 52 (i.e. displaysystem 12, 32, or 42), and the display system 52 is in front of theuser(s) 22. A LAN or WAN 51 is also shown connected to computer system14 for access to information on the internet. The gimbaled sensor system16 tracks the head and eye movements of the user 22 and provides theadvantage of the user 22 not having to wear or attach anything tohim/herself. The overall human machine interface system 10 is able totrack the movement of the user's head 18 and eye 20 thereby allowing theuser 22 to control the display system 52 with his/her eye 20 and otherinput devices 53.

[0077] However, any suitable system for eye and head tracking and eyeand speech controlling the scrolling of a display system may be utilizedfor carrying out the purposes of the present invention which is notlimited to the components ways, and methods of implementation shown inFIG. 3B.

[0078]FIG. 3C shows a detailed hardware block diagram of thehuman-machine interface system 10 for the automatic scrolling and speechcontrol of a display or multimedia system 12. The specific hardware forthis system generally includes a display system 52 which includes adisplay driver 68 which is coupled to a computer system 14. (Here, user22 is shown sitting in front of the display screen 12.) The computersystem 14 is attached to a computer interface card 64 and a computerinterface bus 66 through which it communicates with the display driver68 (i.e. VGA card) and the gimbaled sensor system 16. The gimbaledsensor system 16 is shown to be mounted on top of the display system 12coupled with the computer system 14. The gimbaled sensor system 16 iscontrolled to sense and track the position of the user's head 18, and itis also controlled to sense the position at which a user's eye 20 isgazing at the display system 12. The gimbaled sensor system 16 iscoupled to the computer interface card 64 so that the screen gazemeasurements and head position data is sent to the computer interfacecard 64 which is coupled to a computer interface bus 66 which, in turn,is coupled to a microprocessor 72 to control the display of informationon the display screen 12 depending on whether the user 22 is gazing at ascroll activation region. Speech commands such as “step,” “stop,”“search word,” “page down,” “page up,” “magnify,” may also be used alongwith other input devices 53 for selecting or controlling the computersystem 14 and display 12. Detection of lip movement or some otherindicators, such as, nodding of the head, or waving of hands, may bedetected by image processing or some other means that may be used forselecting or controlling the computer system 14 and display 12.

[0079] Referring to block 173 of FIG. 3C, the gimbaled sensor system 16has camera(s) 26 or photo sensor(s) 74 having optics 76 and zoom controlline 78 for focusing on and obtaining various images of the user's head18 and eye 20. The camera 26 or photo sensor 74 is coupled to thecomputer interface card 64 via a image/photosensor interface 80. Thesignals from image/photo sensor interface 80 are then sent to the to thebuffer memory 86 which is attached to computer interface bus 66. Thegimbaled sensor system 16 also includes a distance range finder 88 tofind the distance (between points E 176 and D 120 of FIG. 4A) from whichthe user's head 18 is to the gimbaled sensor system 16. The distancerange finder 88 can be an ultrasonic range finder (i.e. using soundwaves) for determining the distance between the user's head 18 and thegimbaled sensor system 16. The ultrasonic range finder is attached to arange driver interface 90 that activates the ultrasonic range finder,and the range driver interface 90 is coupled between the range finder 88and a buffer memory 86 in the interface card 64. The distance isdetected and determined, and the value of this distance is sent to thebuffer memory 86. Alternatively, the distance range finder 88 can be alaser range finder. The laser range finder is attached to a range driverinterface 90 that activates the laser range finder, and the range driverinterface 90 is coupled between the range finder 88 and a buffer memory86 to the computer interface bus 66 (i.e. ISA, PCI bus, etc.). Commandsfrom the computer system 14 via the buffer memory 86 control the rangedriver 90 to drive the distance range finder 88 (Other range findingdevices may also be used) to determine the distance between the user'shead 18 and the gimbaled sensor system 16. The distance is thenmeasured, and the value of this distance is sent to the buffer memory86.

[0080] The gimbaled sensor system 16 is attached to the computerinterface card 64 via pan driver 92, and the pan driver 92 applies theappropriate voltage to a pan servo 94 to control the pivotal movement β(beta) 177 (i.e. see FIG. 4A in the XZ phase) in the horizontaldirection. The gimbaled sensor system 16 is also attached to thecomputer interface card 64 via tilt driver 96, and the tilt driver 96applies the appropriate voltage to the tilt servo 98 to control thevertical pivotal scanning movement α (alpha) 175 (i.e. see α 175 in FIG.4A on a plane parallel to the Y axis 192). The pan and tilt drivers 92and 96 each control a respective servo-motor, stepper motor or actuatorthat moves or controls the associated gimbaled sensor system 16. The panand tilt servos 94 and 98 allow movement or rotation of the gimbaledsensor system 16 to track the position of the head 18 of the user 22.The angular position (i.e. pan and tilt angles β 177 and α 175respectively) of the gimbaled sensor system are converted from analogvalues to digital values via analog to digital (“A/D”) converters 100and 102, and the corresponding digital values are sent to the memorybuffer 86. Commanded or desired coordinate position values are sent fromthe computer system 14 via buffer memory 86 as well. Thecommanded/desired position values are converted from digital values toanalog values via digital to analog (“D/A”) converters 104 and 106, andthese analog values are sent to the pan and tilt drivers 92 and 96 tocontrol the corresponding desired angles β 177 and α 175 to position thegimbaled sensor system 16.

[0081] Furthermore, a low power infrared laser or LED 77 is coupled tothe optics 76 and also coupled to the light source driver 79. The lightsource driver 79 provides enough power to drive the laser or LED 77. Thelow power infrared laser or LED 77 is used to provide and place a glinton the user's eye 20 to enhance finding the center of the user's eye 20.

[0082] The gimbaled sensor system also includes a directional microphone108 that can provide voice or speech recognition between the user 22 andthe computer system 18. The microphone 108 is coupled to the computerinterface card 64 via an audio amplifier 110. The audio amplifier 110,in turn, is attached to an audio filter 112 which is coupled to ananalog to digital (“A/D”) converter 114. The audio amplifier 110amplifies the signals received from the microphone 108, and theseamplified signals are filtered by the audio filter 112. The filteredsignals are then converted from analog to digital signals, and thedigital signals are sent to the buffer memory 86.

[0083] Therefore, the computer interface card 64 functions to receivethe relevant information or data relating to the position of the user'shead 18 and eye 20 from the gimbaled sensor system 16 and sends thisinformation and data to the memory buffer 86. The memory buffer 86interfaces with the computer system 14 via a computer interface bus 66,and the computer interface bus 66 is coupled to a display driver 68(i.e. VGA card). The display driver 68 drives the display system 12. Thecomputer system 14 runs the algorithm to control the gimbaled sensorsystem 16 and directs the corresponding hardware to perform desired orcommanded functions via the movement of the user's eye 20 or via theuser's voice or speech commands. For example, a facial recognitionprogram can be executed by the computer system 14 to track the movementof the head 18 of a specific user 22. The program is executed, and ifthe user 22 moves his head 18 to a different position, then the camera26 or photo-sensor 74 picks the images up through the optics 76 andcompares it with the previous position of the user's head 18. The zoomcontrol line 78 is connected via a digital to analog (“D/A”) converter81 to buffer memory 86. The gimbaled sensor system 16 is directed tocorrespondingly follow the movement of the head 18 by sending commandsto the pan and tilt drivers 92 and 96 to control the angles of the panand tilt servos 94 and 98 to move the gimbaled sensor system 16 to aposition in which the gimbaled sensor system 16 is locked onto theuser's eye 20 by following the 3-D head movement.

[0084] However, any suitable hardware or components may be utilized forcarrying out the purposes of the present invention which is not limitedto the above described gimbaled sensor system in FIG. 3C.

[0085] B. Vectors and 3-D Geometry Associated With Eye/Head TrackingSystem

[0086]FIG. 4A shows a perspective view of the vectors and 3-D geometryin Euclidean coordinates that are associated with eye/head tracking.Point A 174 is at the center of the display screen 12 as shown in FIG.4A. Screen gaze point B 146 in 316 D coordinates on the display screen12 is the point at which the user 22 is focused on the display screen12. Point C 118 in FIG. 4A represents the center of the cornea of theuser's eye 20 in 3-D coordinates while point D 120 represents the centerof the glint on the user's eye 20 provided by the infrared laser or LED77 (i.e. see FIG. 3C) in 3-D coordinates with point E 176 at origin ofcoordinate axis X 190, Y 192, and Z 194. The rotational axis which areset to be aligned with coordinate axis X 190 and Y 192 when β=0 of thegimbaled sensor system also intersect at point E 176. Point E 176 isaligned vertically and directly above point A 174 (i.e. the center ofthe display or screen 12). Vectors ED 178, CD 182 (not shown), and CB180 show gimbaled sensor system 16 line of sight as vector ED of theuser's eye 20 and screen gaze direction as vector CB 180, and the vectorbetween glint 119 and cornea 117 centers (see FIG. 4C). Tilt and panservo angles α 175 and β 177 of the gimbaled sensor system 16 (Shown inFIG. 1A) at point E 176 and range distance measurement(s) (by ultrasonicand/or laser range finding equipment) and/or by appropriateimage/processing algorithms) along with R 206 and y 121 are all used tocalculate the screen gaze coordinates at point B 146 in FIG. 4A. The eye20 may be illuminated by a low power infrared laser or LED 77 (i.e. seeFIG. 3C) placed in the center of the camera field of view (i.e. Point E176) to further enhance measurements of points C 118 and D 120 as isknown in the prior art.

[0087] However, any suitable system or method associated with theperformance of eye and head tracking may be utilized for carrying outthe purposes of the present invention which is not limited to the waysand methods shown in FIG. 4.

[0088] C. Determining Closest User

[0089]FIG. 4B shows a wide field of view from camera(s) 26 orimage/photo sensor 74 (shown in FIG. 3C) with three users 184, 186, and188 and Cartesian coordinate axes X 190, Y 192, and Z 194. The closestuser 188 may be detected by first doing an ultrasonic scan of the fieldof view to find the closest object. A segmentation image processingalgorithm may then be used to segment the profile of the closest user188 from a wide camera field of view and segment the user's eyes 20thereby identifying the user's eyes 20 by determining the relative sizeof the segmented region around each of the user's eyes 20. The gimbaledsensor system 16 is then adjusted to focus on the approximate center ofone of the closest user's eyes 20. A magnification step is then appliedin either large discrete steps 202, 200, 198, 196, and 204 orcontinuously until the eye 20 covers up the entire field of view 204 ofthe camera 26 or image/photo sensor 74 as shown in FIG. 4C.

[0090] However, any suitable system or method for determining theclosest user may be utilized for carrying out the purposes of thepresent invention which is not limited to the ways shown in FIG. 4B.

[0091] D. Magnified Field of View of User's Eye

[0092]FIG. 4C shows the magnified field of view 204 of the camera(s) 26or image/photo sensor 74 that is part of the gimbaled sensor system 16.This magnified view 204 is focused on the center of glint, enhanced byinfrared LED or laser 77 as seen in FIG. 3C, of the user's eye 20 atpoint D 120. The gaze-offset of the user's pupil to point C 118 withrespect to point D 120 shown with angle γ (gamma) 121, and R 206 isdetermined by calculating the center of the glint as point D 120 and thecenter of the cornea as point C 118. The angles α 175 and β 177 of FIG.4A, radial distance R 206 and angle γ of FIG. 4C, and the distancebetween point E 176 and point D 120 of FIG. 4A, are used as inputs to aneural network (that is trained by supervised learning) or otherfunction approximator to calculate the screen 12 gaze coordinates atpoint B 146 shown in FIG. 4A.

[0093] However, any suitable system or method for magnifying the fieldof view of a camera or photo sensor may be utilized for carrying out thepurposes of the present invention which is not limited to the ways shownin FIG. 4C.

[0094] B. The Software Algorithm for the Specific Automatic ScrollingSystem

[0095] 1. The Overall Software Algorithm for the Specific AutomaticScrolling System

[0096] FIGS. 5 to 7 show the method and screen sensor coordinatemeasurement algorithm 116 of controlling automatic scrolling and/orcontrol of information on a display or screen 12 by obtaining gazemeasurements for providing the relative position of the user's head 18and eye 20 to the display screen 12 and gimbaled sensor system 16.

[0097]FIG. 5A shows a specific screen gaze coordinate measurementalgorithm 116 that drives the hardware system 10 shown in FIGS. 1A, 1B,1C, 3A, 3B and 3C to control the scrolling and display of information onthe display screen 12. This screen gaze measurement algorithm 116 isused for identifying the points C 118 (cornea 117 center) and D 120(glint 119 center) within the user's eye 20 as shown in FIGS. 4A, 4B,and 4C. The screen gaze measurement algorithm 116 begins at start block122, and the algorithm 116 first finds, at block 124, whether a user 22is detected by using the camera(s) 26 or image/photo sensor 74, andrange finder 88 of FIG. 3C, or directional microphone 108, and imageprocessing (i.e. obtaining and comparing images) method or,alternatively, by an ultrasonic method finding minimum distant. The nextstep in algorithm 116 is to determine at decision block 126 whether theclosest user has been detected. If the closest user 22 has not beendetected, then the algorithm 116 loops back to block 124 to continue tosearch for the closest user and again checks at decision block 126whether the closest user 22 has yet been detected. If a number of users22 are detected by the camera 26 or photo sensor 74 and image processingmethod at block 124, then the closest user 22 is determined by asuitable way for determining the closer of several images (i.e.segmentation image processing and/or ultrasonic scanning) at block 126.

[0098] The next step of the screen gaze measurement algorithm 116 is atdecision block 128 which is to determine whether the camera or photosensor field of view is centered and magnified on the user's eye 20. Ifthe field of view is not centered and magnified on the closest user'seye, then the next steps of the algorithm determine the correctionvector for re-centering the field of view, as shown in block 130, and tozoom or position the camera(s) 26 or image/photo sensor 74 field of viewonto the center of the closest user's eye 20, as shown in block 132where the magnified image is then read in at block 134 where programcontrol is passed via connector A 133. The re-positioning of thecamera(s) 26 or image/photo sensor 74 at the center of the user's eye 20is accomplished via pan and tilt servos 94 and 98 (See FIGS. 3B and 3C),and the magnification of the image of the eye 20 is accomplished throughzoom control line 78 via digital to analog converter 81 of FIG. 3C. There-positioning and magnification results in the field of view of thecamera(s) 26 or image/photo sensor 74 being covered by most of theuser's 22 eye 20. If the field of view is centered and magnified on theclosest user's eye 20, then the algorithm screen gaze coordinatemeasurement 116 moves directly to the next step to determine 2Dcoordinate of points C 118, and D 120 on user's eye 20 at block 136after reading in the magnified image of the user's eye 20 which isanalyzed by image processing.

[0099] The next task to be accomplished by the screen gaze coordinatemeasurement algorithm 116 is to determine the 2-D screen gazecoordinates B(X_(b),Y_(b)) 146 through a perspective mapping given α 175and β 177, distance between points E 176 and D 120, distance betweenpoints D 120 and C 118, andy 121, at block 140. To determine the screengaze coordinates, the screen gaze coordinate algorithm 116 must firstdetermine the 2-D coordinates of the points C 118 and D 120 on theuser's eye 20. There are many methods of calculating the screen gazecoordinates, but because of the inherent non-linearities and uniquenessin the 3D structure of a user's eye 20, these calculations can be verycomplex. One method of calculating the screen gaze coordinatesB(X_(b),Y_(b)) 146 from α 175, β 177, distance ED 178 between points E176 and D 120, distance R 206 between points D 120 and C 118, and angleγ 121 is by using a neural network (see FIG. 5C) as a functionapproximator. Neural networks may be good function approximators ifproperly trained. An appropriate calibration procedure can be used fortraining each user in conjunction with an automatic user identificationsystem that remembers the corresponding training weights for each userthat has been identified. Thus, calibration may not be requiredfrequently if each user is uniquely identified. The training of theneural network can be done through ordinary supervised learning. This isaccomplished by having a calibration procedure where the user is askedto follow a cursor on the screen with their eyes where the error betweenthe network output and the actual cursor position is used to adjust theweights of the neural network. The generalizing capability of the neuralnetwork may also be strong enough to cover all user's eyes 20 dependingon the robustness of the algorithm used and thereby not requiring anycalibration procedure.

[0100] At block 136, the glint 119 and cornea 117 perspective centers D120 and C 118 respectively are obtained by segmenting and clustering themagnified eye image 204 and calculating the center of gravity of the twosmaller clusters representing the glint 119 and cornea 117. After theperspective view of the centers D 120 and C 118 are found in the imageplane, the angle γ 121 and the distance R 206 can be calculated. Afterthe inputs α 175, β 177, distance ED 178 between points E 176 and D 120,R 206 between points D 120 and C 118, and angle γ 121 are measured andcalculated, the screen gaze coordinate measurement algorithm 116 goes onto block 140 to calculate the screen gaze coordinates B(X_(b),Y_(b))146. At block 140, the screen gaze coordinates B(X_(b),Y_(b)) 146 aredetermined through a neural network (see FIG. 5C), and these coordinatesare sent to the software application program at block 142. The algorithmends at block 144.

[0101] Any suitable algorithm, software program, sensing system, orother implementation for approximating the screen gaze coordinatesB(X_(b),Y_(b)) 146 may be utilized for carrying out the purposes of thepresent invention which is not limited to the ways described above andshown in FIG. 5A.

[0102] 2. The Image Processing Algorithm for Head Tracking

[0103]FIG. 5B shows an algorithm 130 for calculating correction vector(i.e. used to determine the closest user 22 and track the user's head 18and eye 20) for recentering (see block 130 of FIG. 5A). The algorithm130 starts at block 148. The first step at block 150 of the algorithm130 is to read in raw image data. The next step at block 152 is tofilter image (i.e. filter out high frequency spatial data such as imagenoise), and at block 154, images are segmented (i.e. edge detection).The algorithm 130 then moves to the next step at block 156 of formingclusters by finding regions that have edges and putting them intogroups, and the following step at block 158 is to determine headclusters (i.e. find oval shaped cluster of medium size). The algorithm130 then goes to the step at block 160 for determining eye regions ofhead cluster (i.e. find the small cluster at the top region of the headcluster and choose the cluster of medium size having the center closestto the image center) and goes on via connector G 162 to the step atblock 168 for determining the area of a user's eye cluster. At block164, the “center of gravity” of the calculated eye cluster iscalculated. The algorithm step at block 166 is to adjust the pan andtilt angles α 175 and β 177, respectively, of the gimbaled sensor system16 to bring the image center to line up with glint 119 center at point D120 of FIG. 4A, 4B, and 4C. The camera(s) 26 or image/photo sensor 74view is then zoomed in so that a large portion of an eye cluster area isin the field of view of the camera(s) 26 or image/photo sensor 74centered on point D 120. The image processing algorithm then ends atblock 172.

[0104] However, any suitable algorithm, software program, or otherimplementation for image processing may be utilized for carrying out thepurposes of the present invention which is not limited to the ways shownin FIG. 5B.

[0105]FIG. 5C shows an example of a neural network algorithm 140 used toapproximate screen gaze coordinates X_(b) 185 and Y_(b) 187 of point B146. Five inputs are shown as α 175, β 177, distance ED 178 from point E176 to point D 120, R 206, and γ 121, all connected to input layer 171with five nodes. A hidden layer 179 with four nodes is shown connectedto the input layer 171 via connecting weights (W_(1x)) 173. Output layer183, shown with two nodes and output screen gaze coordinates 167 ofpoint B 146, is connected to the hidden layer 179 via connection weights(W_(2x)) 181. A sigmoidal output function f(x) 169 is shown with outputequation 167 where the weighted connections from a previous layer aremultiplied by the outputs of the previous layer and summed. That resultis then added to a threshold offset value. The neural network can betrained using back propagation or any suitable training algorithm. Thefollowing references include more details of neural network design:Lippmann, “An Introduction to Computing with Neural Nets,” IEEE ASSPMagazine April 1987, at pp. 4-22; “Special Issue on Neural Networks II:Analysis, Techniques & Applications,” Proceedings of the IEEE, Vol. 78,No. 10, October 1995; and Widrow, Lehr, “30 Years of Adaptive NeuralNetworks: Perceptron, Madaline and Backpropagation,” Proceedings of theIEEE, Vol. 78, No. 9, September 1990, at pp. 1415-1442.

[0106] V. Automatic Control of Scrolling Embodiment Using PeripheralRectangular Activation Regions

[0107] A. Operation of Peripheral Rectangular Activation Regions

[0108]FIG. 6A shows an embodiment for controlling automatic scrolling ofinformation on a display 12 (32, or 42) wherein the embodiment usesrectangular eye control regions 208, 210, 212, and 214 for scrolling thedisplay or screen 12 (i.e. generally for scrolling text but not limitedto text). Four rectangular scroll control regions are shown as follows:an upper region 208, a lower region 210, a right region 212, and a leftregion 214. Within each control region, four scroll rate regions aredefined as follows: a low scroll rate region 216, a medium low scrollrate region 218, a medium high scroll rate region 220, and a high scrollrate region 222 are shown around the peripheral of the display or screen12. The inner scroll regions 216 and 218 have a slower scroll rate thanthe outer scroll regions 220 and 222. The scroll rate can be either aquantized (i.e. discrete scroll rate for each region) or a continuousfunction (i.e. intermittent levels of scroll rate from low to highcorrelated respectively to a position from the inner to the outer partof the region). The scroll rate is determined by the screen gazecoordinate of the user 22 on the screen 12 (i.e. point B 146 as seen inFIG. 4A). The scroll rate is designed to adaptively adjust to thereading speed of the user 22. If a user 22 gazes in the static controlarea 224 of the display or screen 12, then the scroll rate is set tozero.

[0109] B. Examples of Gaze Points Within the Screen or Display withPeripheral Activation Regions

[0110] 1. Gaze Points Within the Activation and Static Regions

[0111] Some example gaze points are shown at P1 226, P2 228, P3 230, andP4 232 of the display 12 as shown in FIG. 6A. As stated earlier, thecenter of the display or screen surface 12 is represented at point A174. If a user 22 gazes at point P1 226 within the medium low scroll upregion 218 and low scroll left region 216, then the text will scroll onthe display or screen 12 to the upper left by scrolling up at a mediumlow rate and to the left at a low rate. If a user 22 gazes at point P2228 at the left side region within the medium high scroll control region220, then the text on the display or screen 12 is scrolled to the rightat a medium high rate. At gaze point P3 230 inside the static controlarea 224, the scroll rate is set to zero and the text does not move. Atgaze point P4 232 inside the high scroll down region 222, the text willscroll down at a high rate (i.e. similar to a “page up” function).

[0112] 2. Gaze Points on Hypertext Link Regions, Pop-Up Menus, andSingle Line Focused Reading

[0113] Hypertext link regions 238 of FIG. 6A are shown in bold italicsand are underlined. For example, if a user dwells on a hypertext region238 at point P5 234, the hypertext region becomes highlighted and theuser may select the hypertext link by a specified selecting protocol, bymeans of a switch, speech commands, or some other type of input device53 (see FIG. 3A,3B or 3C). The new information is then displayed fromthe location specified by the hypertext link that was selected. If thehypertext link region 238 is in a non-static scroll control region, forexample, scroll left region 216, and the user stays focused on thehypertext region 238 for a specified dwell time, as it scrolls to theleft, then the hyper text link may still be selected.

[0114]FIGS. 6A, 6B, 6C, 7A, 7B, and 7C show that a pop-up menu selectionregion 240 can be used with the present scroll system 10 in the top leftarea of the display or screen 12. In FIG. 6A, if a user 22 gazes atpoint P6 236 on pop-up menu selection region 240, then the pop-up menuselection region 240 may become highlighted. The scroll regions 216,218, 220, and 222 overlapping the pop-up menu region 240 may be disabledto prevent scrolling when the user is looking at the pop-up menu region240. The pop-up menu is selected by the user either through dwell timeor other selection protocol such as a speech command. A pop-up menu (notshown) may be displayed where a user can then select through eyecontrol, speech commands, or by other means such as a hand controlledmouse or keyboard. A popup menu may be activated by other means withouthaving a pop-up menu selection region 240. For example, any selectionmade by speech, keyboard, or mouse control.

[0115]FIGS. 6B and 6C show an example of text scrolling if the user wereto gaze at point P 226 with old coordinates X, Y, in FIG. 6B and newcoordinates Xn, Yn in FIG. 6C at the bottom right area of the display orscreen 12. In these figures, the text scrolls upward at a medium lowrate and to the left at a low rate. The pop-up menu selection region240, which may be semi-transparent to underlying information, is shownin the top left area of the display or screen 12 in FIGS. 6A, 6B, and6C. As a further option, the text may be adjusted so that a user 22 canfocus on one line and read through the entire document (i.e. ticker tapestyle) without having to go back to start at the next line or the end ofthe previous line. This can be done by shifting the text up one line ordown one line as a person respectively scrolls the text to the right orleft (i.e. the next screen to the right would be identical to thepresent screen except shifted up one line while the previous screen tothe left would be identical to the present screen except shifted downone line).

[0116] C. Scrolling Algorithm for Rectangular Peripheral Control Regions

[0117]FIG. 6D shows the gaze controlled scrolling algorithm 242 usingrectangular peripheral control regions 208, 210, 212, and 214 outlinedin flow chart form. The algorithm 242 starts at block 246. At block 248,the eye gaze screen coordinates (X, Y) are obtained as described inFIGS. 4A, 4B, 4C, 5A, 5B, and 5C. The algorithm 242 then moves todecision block 250 where the algorithm 242 checks to see whether theuser 22 is gazing at the pop-up menu control region 240 (shown in FIGS.6A, 6B, and 6C). If the gaze is in this menu region 240, then a decisionat block 252 is made for selection. If the pop-up-menu is selected, thencontrol is passed to the pop-up menu algorithm at block 254, and afterthe pop-up menu algorithm is run, the algorithm 242 passes control tothe end at block 278 through connector D 280 and if pop-up menu region240 is not selected, the pop-up menu routine is bypassed and control ispassed onto the algorithm end 278 via connector D 280. If the eye gazeis not in the pop-up menu region 240, then control is passed to the nextdecision block 256 where hypertext gaze is determined. If the user 22 isnot gazing at a hypertext region 238 (shown in FIGS. 6A, 6B, and 6C),then control of the algorithm 242 passes via connector F 282 todetermine if a gaze is in a scroll region. If the user 22 is gazing at ahypertext region 238 (shown in FIGS. 6A, 6B, and 6C), then the region238 may become highlighted and control is passed to decision block 258where user selection is determined. User selection can be either dwelltime, key on a keyboard, mouse button, speech recognition, or any otherselection protocol. If no selection is made of the hypertext region 238,then control of the algorithm 242 passes to determine the gaze positionand gaze region of the user 22 via connector F 282. If a hypertextregion 238 selection, however, is made, then the algorithm 242 passes toblock 260 where hypertext link is activated and new information is shownfrom the location specified by the hypertext link 238 that was selected(i.e. jump to hypertext location), and the algorithm 242 then passescontrol to the end at block 278 through connector D 280.

[0118] The algorithm 242 passes control to determine gaze position andgaze region of the user 22 via connector F 282. At decision block 262,the algorithm 242 determines whether the user's gaze is in upper controlregion 208 (see FIG. 6A). If the gaze is in this upper region 208, then,at block 264, the text is scrolled down at a rate dependent upon thegaze position within the control region 208 (i.e. inside a low 216,medium low 218, medium high 220, and high 222 region, within the upperregion 208 as shown in the upper part of FIG. 6A). The algorithm 242then passes control to decision block 266 to determine whether the gazeis in the lower control region 210 (i.e. FIG. 6A). If the gaze is in thelower control region 210, then, at block 268, the text is scrolled up ata rate dependent upon the screen gaze position within the control region210 (i.e. inside a low 216, medium low 218, medium high 220, and high220 region, within the lower region 210 as shown in the lower part ofFIG. 6A). The algorithm 242 further passes control to decision block 270where the algorithm 242 determines if the gaze is in the right controlregion 212 (see FIG. 6A). If the gaze is in the right control region212, then, at block 272, the text is scrolled to the left at a ratedependent upon the gaze position within the control region 212 (i.e.inside a low 216, medium low 218, medium high 220, and high 222 region,within the right region 212 as shown in the right part of FIG. 6A). Thealgorithm 242 then passes control to decision block 274 where thealgorithm 242 determines if the gaze is in the left control region 214(see FIG. 6A). If the gaze is in the left control region 214, then thetext is scrolled to the right at a rate dependent upon the gaze positionwithin the control region 214 (i.e. inside a low 216, medium low 218,medium high 220, and high 222 region, within the left region 214 asshown in the left part of FIG. 6A).

[0119] However, any suitable algorithm or method for controlling thescrolling of information, data, text, images, etc. on a screen ordisplay by using rectangular peripheral control regions may be utilizedfor carrying out the purposes of the present invention which is notlimited to the ways shown in FIG. 6A, 6B, 6C, and 6D.

[0120] V. Automatic Control of Scrolling Embodiment Using ActivationRegions Defined by Concentric Rings

[0121] A. Operation of Quadrant Activation Regions Defined by ConcentricRings

[0122]FIG. 7A shows circular eye control regions for scrolling thedisplay or screen 12 (i.e. generally for scrolling images but notlimited to images). Four regions separated by horizontal axis 302 andvertical axis 304 are shown as follows: an upper right quadrant region I290, a lower right quadrant region II 292, a lower left quadrant regionIII 294, and an upper left quadrant region IV 296. The scroll controlregions low 310, medium 308, and high 306 are respectively shown inconcentric rings about the screen 12 and screen center point A 174. Therings 310, 308, and 306 are concentric about the screen 12 center pointA 174. The scroll control regions 310, 308, and 306 can respectively beeither quantized or discrete levels of low, medium, or high scroll ratesas shown in FIG. 7A or the scroll rate and direction can be controlledas a function of the screen gaze angle φ (phi) 300 and distance D 298(distance between A 174 and P 288) repeating as described later. Staticzones or regions are defined as the inner most circle(s) 318, etc. andregion(s) 320 beyond the outer most ring 316. The inner scroll regions310 has a slower scroll rate than scroll region 308, and scroll region308 has a slower scroll rate than scroll region 306. The scroll rate maybe either a quantized (i.e. discrete scroll rate for each region) or acontinuous function (i.e. intermediate levels of scroll rate from low tohigh correlated respectively to a position from the inner to the outerpart of the region). The scroll rate is determined by the screen gazeposition of the user 22 on the screen or display 12 within a quadrantand within a scroll rate concentric ring region.

[0123] B. Example of Activation Regions Defined by Concentric Rings

[0124] In FIG. 7A, a map of the United States 284 is displayed on thedisplay or screen 12. The state of Michigan at point A 174 is centeredon the display or screen 12. A gaze point P 288 of the user 22 is shownon the bottom right of the display or screen 12 in quadrant II 292 atthe top of the state of Florida (partially occluded by screen edge). Avector D 298 is indicated on FIG. 7A, which represents the directionfrom point A 174 to point P 288 with a negative angle φ 300 from thehorizontal axis 302 as shown in FIG. 7A. (A pop-up menu selection region240, which may be semi-transparent to information displayed underneath,is also shown in quadrant IV 296, and the menu selection region 240 canbe highlighted and/or selected when gazed upon by the user 22.

[0125]FIGS. 7B and 7C show an example of image scrolling byconcentric-ring scroll control regions 306, 308, or 310 of FIG. 7A. InFIG. 7B, if a user 22 gazes at the point P 288, which is inside the highscroll rate control region 306 of FIG. 7A at screen coordinates X and Y,the image scrolls up and to the left at a rate calculated from thevector R shown in FIG. 7A. The new position of point P 288 is shown inFIG. 7C with new screen coordinates Xn and Yn. The equations withproportionality constant K_(x) and K_(y) for adjusting scroll speeds areshown below:

D=SQRT[((X _(a) −X)²+(Y _(a) −Y)²)]  Eq.(1)

φ= TAN⁻¹[(Y−Y _(a))/(X−X _(a))]  Eq.(2)

X Scroll Rate=K _(x) *D*COS(φ)  Eq.(3)

Y Scroll Rate=K _(y) *D*SIN(φ)  Eq.(4)

[0126] The scroll rates may be quantized to the levels of low (i.e.region 310), medium (i.e. region 308), and high (i.e. region 306) byquantizing D 290 to those levels. A transparent pop-up menu selectionregion 240 is shown in the upper left corner of FIGS. 7A, 7B, and 7C.

[0127] C. Scrolling Algorithm for Concentric, Ring, Control Regions

[0128]FIG. 7D shows a flow chart of the algorithm 322 for the circularscroll control regions 306, 308, and 310 shown in FIG. 7A. The algorithm322 begins at block 324. At block 326, eye gaze screen coordinates X andY are measured and calculated. The algorithm 322 passes control todecision block 328 to determine if the user 22 is gazing at the pop-upmenu selection region 240. If the user 22 is gazing at this region 240,then it may be highlighted, and the algorithm 322 then passes control todecision block 330 to determine if the pop-up menu has been selected. Ifthe pop-up menu 240 has been selected either by dwell time or some otherselection protocol, then a pop-up menu algorithm is run at block 332 andthen ends at block 344 via connector E 342. If the pop-up menu 240 isnot selected, then the algorithm 322 ends at block 344 via connector E342. If the user's gaze is outside of pop-up menu region 240, thenalgorithm control is passed onto decision block 334 to determine if theuser's gaze is inside the inner static region 318 or the outer staticregion 320 of the display or screen 12 of FIG. 7A. If the user's gaze isin a static zone 318 or 320, then the algorithm 322 ends at block 344via connector E 342. If the user's gaze is not inside a static region318 or 320, then the radius and angle of the gaze point are calculatedat block 336 as described by Eq. (1) and Eq. (2) respectively. At block338, the horizontal (X) and vertical (Y) scroll rates are calculated byusing Eq.(3) and Eq.(4) respectively. The image is scrolled in thecalculated horizontal and vertical rates at block 340 where thealgorithm 322 ends at block 344.

[0129] However, any suitable algorithm or method for controlling thescrolling of information, data, text, images, etc. on a screen ordisplay 12 by using concentric, ring scroll control regions may beutilized for carrying out the purposes of the present invention which isnot limited to the ways shown in FIGS. 7A, 7B, 7C, and 7D. Other shapesof control regions may be used for the purpose of scrolling.

[0130] VI. Scope of Disclosure

[0131] The preferred embodiment of this invention is described above inthe Figures and Detailed Description. Unless specifically noted, it isthe intention of the inventors that the words and phrases in thespecification and claims be given the ordinary and accustomed meaningsto those of ordinary skill in the applicable art(s). The foregoingdescription of a preferred embodiment and best mode of the inventionknown to applicant at the time of filing the application has beenpresented for the purposes of illustration and description. It is notintended to be exhaustive or to limit the invention to the precise formdisclosed, and many modifications and variations are possible in thelight of the above teaching. The embodiment was chosen and described inorder to best explain the principles of the invention and its practicalapplication, and to enable others skilled in the art to best utilize theinvention in various embodiments and with various modifications as aresuited to the particular use contemplated.

[0132] Likewise, the use of the words “function” or “means” in theDetailed Description is not intended to indicate a desire to invoke thespecial provisions of 35 U.S.C. Sec. 112, Paragraph 6 to define hisinvention. To the contrary, if the provisions of 35 U.S.C. Sec. 112,Paragraph 6 are sought to be invoked to define the invention, the claimswill specifically state the phrases “means for” or “step for” and afunction, without reciting in such phrases any structure, material, oract in support of the function. Even when the claims recite a “meansfor” or “step for” performing a function, if they also recite anystructure, material, or acts in support of that means or step, then theintention is not to invoke the provisions of 35 U.S.C. Sec. 112,Paragraph 6. Moreover, even if the inventors invoke the provisions of 35U.S.C. Sec. 112, Paragraph 6 to define the invention, it is theintention that the invention not be limited only to the specificstructure, material, or acts that are described in his preferredembodiment. Rather, if the claims specifically invoke the provisions of35 U.S.C. Sec. 112, Paragraph 6, it is nonetheless the intention tocover and include any and all known or later developed structures,materials, or acts that perform the claimed function, along with any andall known or later developed equivalent structures, materials, or actsfor performing the claimed function.

[0133] For example, the present invention specifically makes referenceto hardware relating to the automatic scroll control system thatincludes a computer system, displays, screens, or monitors, othercomputer peripherals, a head tracking system, an eye tracking system, ahead-mounted screen or display, and a transparent non-attached displayor screen. However, numerous other types of computer systems, displays,screens, monitors, head tracking systems, eye tracking systems,head-mounted screens or displays, transparent non-attached displays orscreens are well known to exist, and most likely, numerous othercomputer and tracking related systems, devices, and components will bedeveloped in the future. The inventions described herein are not to belimited in use with the specifically referenced types of computer andtracking related systems, devices, components, and peripherals, butrather, are intended to be used with any and all types of computer andtracking related systems, devices, components, and peripherals.

[0134] As another example, the present invention specifically makesreference to other hardware relating to filters, amplifiers, converters,distance range finder, camera, photo sensor, tilt drivers, servos,microphone, that are used to make up the automatic scroll controlsystem. The disclosure specifically references several examples of suchcomponents, including laser or ultrasonic range finders, pan and tiltdrivers, and pan and tilt servos, etc. However, numerous other hardwarecomponents for an automatic scroll control system are well known toexist, and, most likely, numerous hardware components for such a systemwill be developed in the future. The inventions described herein are notto be limited to the specific components or sub-systems disclosed in thepreferred embodiment, but rather, are intended to be used with any andall applicable automatic scroll control systems. Likewise, the preferredembodiment depicted in the drawings show an automatic scroll controlsystem with various components. Numerous other configurations, andmultiple automatic scroll control systems, can be substituted for thesingle device.

[0135] Furthermore, the present invention specifically makes referenceto a display or screen. However, any other equivalently defined displaysor screens such as a windows, menus, etc. or information defined byboundaries are well known to exist, and, most likely, numerous othersuch displays or screens will be developed in the future. The inventionsdescribed herein are not to be limited in use with the specificallyreferenced types of displays or screens, but rather, are intended to beused with any and all types of displays, screens, windows, menus, or anyother structures, methods, or boundaries that provide a display, screen,window, menu, sub-display, sub-screen environment, etc.

[0136] Further, the present invention specifically makes reference to analgorithm for obtaining head and eye gaze measurements for providing therelative position of the user's head and eye to the display or screen.However, numerous other algorithms or steps for such a method are wellknown to exist, and, most likely, numerous algorithms or steps for sucha method will be developed in the future. Additionally, the presentinvention specifically makes reference to image processing methods orother methods for tracking the user's head and eye or for determiningthe closest user. However, numerous other algorithms or steps for suchmethods are well known to exist, and, most likely, numerous methods forsuch purposes will be developed in the future. The inventions describedherein are not to be limited to the specific algorithms, methods, orsteps disclosed in the preferred embodiment, but rather, are intended tobe used with any and all such methods, algorithms, or steps. In itspreferred form, applicant divides the method for obtaining head and eyegaze measurements for providing the relative position of the user's headand eye to the display or screen into several steps. However, withappropriate knowledge and application of that knowledge to those ofordinary skill in the art, some of the steps can be implemented into asingle step. Likewise, applicant divides the method of image processingor other methods for tracking the user's head and eye for determiningthe closest user into several steps. However, with appropriate knowledgeand application of that knowledge to those of ordinary skill in the art,some of the steps can be implemented into a single step. Thus, it is notthe intention to limit the invention to any particular form or anynumber of method steps or to any specific procedural arrangement.

[0137] Also, the present invention specifically makes reference to anembodiment of a rectangular peripheral control regions and anotherembodiment of a quadrant concentric ring peripheral control regions forcontrolling the automatic scrolling of information on a display orscreen. The present invention further discloses various scroll ratecontrol regions such as quantized scroll rate regions (i.e. discretescroll rates) or a continuous scroll rate function to allow the user tocontrol the rate of scrolling. However, numerous other scroll controlways or scroll rate control methods either exist or are well known toexist. The inventions described herein are not to be limited to thespecific scroll control or scroll rate control methods disclosed in thepreferred embodiment, but rather, are intended to be used with and allsuch scroll control or scroll rate control methods.

[0138] Furthermore, the present invention specifically makes referenceto a number of applications for the system for controlling automaticscrolling of information on a display or screen such as scroll controlfor a computer system, head-mounted display or pair of display glassessuch as those used by technicians or assembly line workers, atransparent non-attached screen or display such as heads-up display ofmedical information for medical personnel and doctors. However, numerousother applications for the automatic scroll control system are wellknown to exist, and, most likely, other applications will be developedin the future. The inventions described herein are not to be limited tothe applications for the automatic scroll control system disclosed inthe preferred embodiment, but rather, are intended to be used with andall such scroll control or scroll rate control methods.

What is claimed is:
 1. A system for controlling automatic scrolling of information on a display or a screen comprising: a computer system coupled to the display or the screen, an eye scroll control sensor coupled to the computer system for tracking and determining a position of the eye of a user relative to the display or the screen, and a scroll activating interface system coupled to the computer system and interfaced with the eye scroll control sensor for implementing automatic scrolling based upon the position of the eye of the user relative to an activation area on the display or the screen.
 2. The system for controlling automatic scrolling of information according to claim 1 wherein the eye scroll control sensor further comprises: an eye tracking system for tracking the eye of the user, and an eye position determining system for determining the position of the eye of the user relative to the display or the screen.
 3. The system for controlling automatic scrolling of information according to claim 1 wherein the eye scroll control sensor is an eye and head scroll control sensor coupled to the computer system for tracking and determining the position of the eye of a user and a position of the head of the user relative to the display or the screen.
 4. The system for controlling automatic scrolling of information according to claim 3 wherein the eye and head scroll control sensor further comprises: a gimbaled head sensor system for tracking the position of the head of the user relative to the display or the screen.
 5. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises: a camera or photo sensor for providing images of the head and the eye of the user, a zoom lens coupled to the camera or photo sensor for focusing the camera or photo sensor at the user, and optics coupled to the camera or photo sensor for aiding the camera or photo sensor in detecting or providing images of the head and the eye of the user so that the images can be processed by the computer system.
 6. The system for controlling automatic scrolling of information according to claim 5 wherein: the computer system further comprises a computer interface card, and the computer interface card further comprises a digital to analog converter coupled to the zoom lens for converting a digital control signal from the computer system to an analog control signal for focusing the camera or photo sensor at the user.
 7. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises: an LED or laser coupled to optics of the gimbaled targeted at the head and eye of the user to provide a glint on the eye of the user to enhance focus of the gimbaled onto a center of the eye of the user.
 8. The system for controlling automatic scrolling of information according to claim 7 wherein: the computer system further comprises a computer interface card, and the computer interface card has a light source driver for driving the LED or laser that is coupled to optics of the gimbaled head sensor system.
 9. The system for controlling automatic scrolling of information according to claim 4 wherein the gimbaled head sensor system further comprises: a distance range finder for determining a distance between the display or the screen and the user.
 10. The system for controlling automatic scrolling of information according to claim 9 wherein the distance range finder is an ultrasonic range finder for determining the distance between the display or the screen and the user.
 11. The system for controlling automatic scrolling of information according to claim 9 wherein the distance range finder is an LED or laser range finder for determining the distance between the display or the screen and the user.
 12. The system for controlling automatic scrolling of information according to claim 9 wherein the computer system further comprises a computer interface card, and the computer interface card has a range driver that drives the distance range finder.
 13. The system for controlling automatic scrolling of information according to claim 4 wherein: the gimbaled head sensor system further comprises a directional microphone for detecting or sensing audio signals from the user to provide voice or sound control to the user of the automatic scrolling, the computer system further comprises a computer interface card, and the computer interface card further comprises: an audio amplifier coupled to the directional microphone for amplifying the detected or sensed voice or sound signals, and an audio filter coupled to the audio amplifier for filtering out any noise or interference that is detected or sensed by the directional microphone.
 14. The system for controlling automatic scrolling of information according to claim 13 wherein: the computer interface card further comprises an analog to digital converter coupled to the audio filter for converting the audio signals that are audio analog signals to audio digital signals so that the audio digital signals are capable of being processed by the computer system.
 15. The system for controlling automatic scrolling of information according to claim 4 wherein: the gimbaled head sensor system further comprises a camera or photo sensor for detecting or providing video images of the head and the eye of the user to provide video signals to be image processed by the computer system so that the head sensor camera or photo sensor is focused on a center of the eye of the user, the computer system further comprises a computer interface card, and the computer interface card further comprises: a video amplifier coupled to the camera or photo sensor for amplifying the video signals, and a video filter coupled to the video amplifier for filtering out any video noise or video interference that is picked up or sensed by the camera or photo sensor.
 16. The system for controlling automatic scrolling of information according to claim 15 wherein: the computer interface card further comprises an analog to digital converter coupled to the video filter for converting the video signals that are analog video signals to digital video signal so that the digital video signals are capable of being processed by the computer system.
 17. The system for controlling automatic scrolling of information according to claim 4 further comprising a pan servo and a tilt servo coupled to the gimbaled head sensor system for respectively controlling the horizontal and vertical pivotal movements of the gimbaled head sensor system.
 18. The system for controlling automatic scrolling of information according to claim 17 wherein: the computer system further comprises a computer interface card, and the computer interface card further comprises: an analog to digital converter coupled to the pan servo for converting analog horizontal position signals of the gimbaled head sensor system to digital horizontal position signals, and an analog to digital converter coupled to the tilt servo for converting analog vertical position signals of the gimbaled head sensor system to digital vertical position signals.
 19. The system for controlling automatic scrolling of information according to claim 17 wherein: the computer system further comprises a computer interface card, and the computer interface card further comprises: a pan driver coupled to the pan servo for driving the pan servo and for controlling the horizontal movement of the gimbaled head sensor system, and a tilt driver coupled to the tilt servo for driving the tilt servo and for controlling the vertical movement of the gimbaled head sensor system.
 20. The system for controlling automatic scrolling of information according to claim 19 wherein: the computer interface card further comprises: a digital to analog converter coupled to the pan driver for converting a digital control signal from the computer system to an analog control signal for controlling the pan servo and horizontal position of the gimbaled head sensor system.
 21. The system for controlling automatic scrolling of information according to claim 19 wherein: the computer interface card further comprises: a digital to analog converter coupled to the tilt driver for converting a digital control signal from the computer system to an analog control signal for controlling the tilt servo and vertical position of the gimbaled head sensor system.
 22. The system for controlling automatic scrolling of information according to claim 4 wherein the computer system further comprises: a computer interface bus, a microprocessor coupled to the computer interface bus, memory coupled to the microprocessor, a computer interface card interfaced with the computer interface bus, and a display driver coupled to the computer interface bus and is adapted to be coupled to the display or the screen wherein the display driver drives the display or the monitor.
 23. The system for controlling automatic scrolling of information according to claim 22 wherein the computer interface card further comprises: buffer memory that couples to the computer interface bus and that stores and allows organized providing of digital information between various components of the system.
 24. A method of controlling automatic scrolling of information on a display or a screen by a user comprising the steps of: finding a screen gaze direction on the display or the screen of the user, determining whether the gaze direction is within at least one activated control region, and activating scrolling to provide a desired display of information when the gaze direction is within the at least one activated control region.
 25. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of finding a gaze direction on the display or the screen further comprises the steps of: detecting a user that is closest to the display or the screen, focusing and magnifying a field of view of a camera on an eye of the user to provide a magnified image of the eye, inputting the magnified image into a computer system, determining physical coordinates of a center of a cornea of the eye and a glint center of the eye, determining a vector between the center of the cornea and a glint center on the display or the screen, calculating gaze coordinates of the user on the display or the screen, and sending the gaze coordinates to the computer system for processing by an application program for controlling the scrolling of information on the display or the screen.
 26. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of focusing and magnifying a field of view of a camera on an eye of the user further comprises the steps of: determining a correction vector for re-centering the camera to adjust the field of view, and zooming the field of view of the camera onto the center of the eye of the user.
 27. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of detecting a user that is closest to the display or the screen further comprises: reading raw image data of an image of the user into the computer system, filtering the image of the user, segmenting the raw image data, forming clusters of the raw image data, determining head clusters from the clusters of the raw image data, determining eye regions of the head clusters, determining an eye center of gravity for each of the eye regions, adjusting a pan and a tilt of a camera for bringing an image center to the eye center, calculating an area for the eye cluster, and zooming the camera so that a large portion of an eye cluster is within a field of view of the camera.
 28. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within an upper horizontal region, and scrolling information downwards at a rate based on the screen gaze coordinates that are within the lower horizontal region.
 29. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a lower horizontal region, and scrolling information upwards at a rate based on the screen gaze coordinates that are within the lower horizontal region.
 30. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a right vertical region, and scrolling information leftwards at a rate based on the screen gaze coordinates that are within the right vertical region.
 31. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a left vertical region, and scrolling information rightwards at a rate based on the screen gaze coordinates that are within the left vertical region.
 32. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within an upper horizontal region and a right vertical region, and scrolling information respectively downwards and leftwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the right vertical region.
 33. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within an upper horizontal region and a left vertical region, and scrolling information respectively downwards and rightwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the left vertical region.
 34. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a lower horizontal region and a right vertical region, and scrolling information respectively upwards and leftwards at a rate based on the screen gaze coordinates that are within the upper horizontal region and the right vertical region.
 35. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a lower horizontal region and a left vertical region, and scrolling information respectively upwards and rightwards at a rate based on the screen gaze coordinates that are within the lower horizontal region and the left vertical region.
 36. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within an upper horizontal region and a right vertical region, and allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the right vertical region.
 37. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within an upper horizontal region and a left vertical region, and allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the left vertical region.
 38. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a lower horizontal region and a right vertical region, and allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the upper horizontal region and the right vertical region.
 39. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a lower horizontal region and a left vertical region, and allowing information on the display or the screen to remain stationary when the screen gaze coordinates are within the lower horizontal region and the left vertical region.
 40. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 wherein the step of determining whether the gaze direction is within at least one activated control region further comprises the steps of: determining whether the gaze direction is within a static region defined by at least one of a number of concentric circles, determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles, and activating scrolling according to the gaze direction within the activated control region to provide a desired display of information so that the region at which the gaze direction of the user is directed to the center of the display or the screen at a rate that is defined for the concentric circle at which the gaze direction of the user is directed.
 41. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the steps of: calculating a radius and an angle with respect to a center of the number of concentric circles to define a gaze vector, calculating horizontal and vertical scroll rates based on the gaze vector, and scrolling the information on the display or the screen in the horizontal and the vertical directions based on the calculated horizontal and vertical scroll rates.
 42. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of: determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved downward and leftward.
 43. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of: determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved upward and leftward.
 44. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of: determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved upward and rightward.
 45. The method of controlling automatic scrolling of information on a display or a screen according to claim 40 wherein the step of determining whether the gaze direction is within an activated control region defined by another of the number of concentric circles further comprises the step of: determining whether the gaze direction is within an activated control quadrant wherein the information on the display or the screen is moved downward and rightward.
 46. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of: allowing the user to focus on one line of the information and read an entire document by shifting the information one line up as the user scrolls the information respectively to the right and by shifting the information one line down as the user scrolls the information to the left.
 47. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of: determining whether the gaze direction is within a pop-up menu region, and activating a pop-up menu algorithm when the gaze direction is within the pop-up menu region.
 48. The method of controlling automatic scrolling of information on a display or a screen according to claim 24 further comprising the steps of: determining whether the gaze direction is within a hypertext location, and activating a hypertext switching algorithm for jumping to a selected hypertext location when the gaze direction is within a desired hypertext area.
 49. A method of making a system for controlling automatic scrolling of information on a display or a screen comprising the steps of: providing a computer system coupled to the display or the screen, coupling an eye scroll control sensor to the computer system for tracking and determining a position of the eye of a user relative to the display or the screen, and coupling a scroll activating interface system to the computer system and interfacing the scroll activating interface system with the eye scroll control sensor for implementing automatic scrolling based upon the position of the eye of the user relative to an activation area on the display or the screen.
 50. The method of making a system for controlling automatic scrolling of information according to claim 49 wherein the step of coupling an eye scroll control sensor further comprises the steps of: coupling an eye tracking system to the computer system for tracking the eye of the user, and coupling an eye position determining system to the computer system for determining the position of the eye of the user relative to the display or the screen.
 51. The method of making a system for controlling automatic scrolling of information according to claim 49 wherein the step of coupling an eye scroll control sensor further comprises the step of: coupling an eye and head scroll control sensor to the computer system for tracking and determining the position of the eye of a user and a position of the head of the user relative to the display or the screen.
 52. The method of making a system for controlling automatic scrolling of information according to claim 51 wherein the step of coupling an eye and head scroll control sensor further comprises the step of: coupling a gimbaled head sensor system to the computer system for tracking the position of the head of the user relative to the display or the screen.
 53. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling a gimbaled head sensor system further comprises the step of: coupling an LED or laser to optics of the gimbaled head sensor system targeted at the head and eye of the user to provide a glint on the eye of the user to enhance focus of the gimbaled head sensor system onto a center of the eye of the user.
 54. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling a gimbaled head sensor system further comprises the step of: coupling a distance range finder to the computer system for determining a distance between the display or the screen and the user.
 55. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling the gimbaled head sensor system further comprises the step of: coupling a directional microphone to the computer system for detecting or sensing audio signals from the user to provide voice or sound control to the user of the automatic scrolling.
 56. The method of making a system for controlling automatic scrolling of information according to claim 52 wherein the step of coupling the gimbaled head sensor system further comprises the step of: coupling a camera or photo sensor to the computer system for detecting or providing video images of the head and the eye of the user to provide video signals to be image processed by the computer system so that the head sensor camera or photo sensor is focused on a center of the eye of the user.
 57. The method of making a system for controlling automatic scrolling of information according to claim 52 further comprising the steps of: coupling a pan servo and a tilt servo to the gimbaled head sensor system for respectively controlling the horizontal and vertical pivotal movements of the gimbaled head sensor system.
 58. The method of making a system for controlling automatic scrolling of information according to claim 57 further comprising the steps of: coupling a pan driver to the pan servo for driving the pan servo and for controlling the horizontal movement of the gimbaled head sensor system, and coupling a tilt driver to the tilt servo for driving the tilt servo and for controlling the vertical movement of the gimbaled head sensor system.
 59. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides medical heads-up information on the display or the screen and the eye of the user controls the automatic scrolling of the information.
 60. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to a technician on the display or the screen and the eye of the technician controls the automatic scrolling of the information.
 61. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an assembly line worker on the display or the screen and the eye of the worker controls the automatic scrolling of the information.
 62. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an air traffic controller on the display or the screen and the eye of the controller controls the automatic scrolling of the information.
 63. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information to an assembly line worker on the display or the screen and the eye of the worker controls the automatic scrolling of the information.
 64. The system for controlling automatic scrolling of information according to claim 1 wherein the computer system provides information for an weather person on the display or the screen and the eye of the weather person controls the automatic scrolling of the information.
 65. The method of controlling automatic scrolling of information on a display or a screen according to claim 25 wherein the step of calculating gaze coordinates further comprises the step of: using a neural network to calculate the gaze coordinates on the display or the screen. 